ci: move to Nix (simpler, easy to maintain version) #1320

storopoli · 2024-02-04T14:47:11Z

Description

This is a simpler version of #1257.
Although not fully reproducible (we are fetching Cargo dependencies instead of declaring all of them).
It is WAY easier to maintain and provides most of the benefits and Dev-Experience from Nix.

You can switch flawlessly from Rust stable, MSRV, and nightly using either:

nix develop .: latest Rust stable
nix develop .#stable: latest Rust stable
nix develop .#msrv: Rust MRSV
nix develop .#nightly: Rust Nightly

It handles all the annoyances of Cargo.locks and cargo updates.

Also you don't need to worry about installing binaries from bitcoind or electrs crates,
since we fetch these from nixpkgs (with pinned versions) and add them to your local environment.
Additionally, all the extra dependencies that you need for WASM are handled as well.
Finally, no need to set ENV vars: they are also set all by default to you.

If you are committing typos, or failing to adhere to our guidelines in CONTRIBUTING.md;
we won't let you commit or push.
This is handled by Nix's automated pre-commit hooks which gives helpful error messages.

Please check the Nix.md file to learn more about Nix and what we are aiming to do with the Nix integration into BDK.

I am taking the liberty of tagging certain people that are Nix enthusiast to help the discussion and review.
Cc @notmandatory, @dpc, @casey-bowman, @plebhash, @matthiasdebernardini, @gudnuf, @oleonardolima.

Notes to the reviewers

Pre-commit checks for everything that we enforce:
- Checks for typos in the source code and documentation.
- Checks if all the commits are GPG-signed and follow the conventional commits style.
  (again, please check CONTRIBUTING.md)
- Runs cargo clippy in all workspace.
- Runs cargo fmt in all workspace.
- Runs nixpgs-fmt in all workspace (for .nix files)
Instructions and rationale in Nix.md

Closes #1162. Superseds #1122, #1156, #1165, #1257.

Pinneds Dependencies:

bitcoind: pinned to 0.27.0 in nixpkgs version 24.05.
electrs: pinned to Blockstream's esplora.

Changelog notice

We moved to Nix-based CI and Local development environment to better onboard new developers and enhance the productivity of developers.

Checklists

All Submissions:

I've signed all my commits
I followed the contribution guidelines
I ran cargo fmt and cargo clippy before committing

New Features:

I've added tests for the new feature
I've added docs for the new feature

Bugfixes:

This pull request breaks the existing API
I've added tests to reproduce the issue which are now passing
I'm linking the issue being fixed by this PR

dpc · 2024-02-04T19:04:47Z

This can be fixed with -- --test-threads=2. Shall we add?

Why not increase bitcoind's rpc threads and work queue from the potato-like defaults instead?

dpc · 2024-02-04T19:08:12Z

So just the dev shells, no crane etc.? Seems reasonable. Dev shells are like 80% of benefits for very little Nix involvment.

crates/bdk/Cargo.toml

storopoli · 2024-02-04T19:39:11Z

This can be fixed with -- --test-threads=2. Shall we add?

Why not increase bitcoind's rpc threads and work queue from the potato-like defaults instead?

Good call I did this in a1997cf

bitcoind_conf.args.push("-rpcworkqueue=1024");
bitcoind_conf.args.push("-rpcthreads=64");

evanlinjin · 2024-02-04T19:50:08Z

This can be fixed with -- --test-threads=2. Shall we add?

Why not increase bitcoind's rpc threads and work queue from the potato-like defaults instead?

Good call I did this in a1997cf
bitcoind_conf.args.push("-rpcworkqueue=1024");
bitcoind_conf.args.push("-rpcthreads=64");

64 threads is way too much. I think the ideal is to have 1 thread per CPU core. I think a good compromise is 4 (between desktop and CI).

storopoli · 2024-02-04T19:55:39Z

This can be fixed with -- --test-threads=2. Shall we add?

Why not increase bitcoind's rpc threads and work queue from the potato-like defaults instead?

Good call I did this in a1997cf
bitcoind_conf.args.push("-rpcworkqueue=1024");
bitcoind_conf.args.push("-rpcthreads=64");
64 threads is way too much. I think the ideal is to have 1 thread per CPU core. I think a good compromise is 4 (between desktop and CI).

4 is the default. So no change then? I can easily delete a1997cf (that's why I did it in a separate commit 😄)

dpc · 2024-02-04T23:07:24Z

64 threads is way too much. I think the ideal is to have 1 thread per CPU core. I think a good compromise is 4 (between desktop and CI).

Its a io bound workload. Absent async capabilities default 4 is laughable even for one core and I don't even know why bitcoind is doing it.

evanlinjin · 2024-02-05T03:58:12Z

64 threads is way too much. I think the ideal is to have 1 thread per CPU core. I think a good compromise is 4 (between desktop and CI).

Its a io bound workload. Absent async capabilities default 4 is laughable even for one core and I don't even know why bitcoind is doing it.

I didn't realize it was io-bounded. However, in our test environment, I doubt there will be 64 concurrent calls. How about we meet in the middle and say 16?

dpc · 2024-02-05T04:56:09Z

I was on my phone before, so let me elaborate.

I didn't realize it was io-bounded. However, in our test environment, I doubt there will be 64 concurrent calls. How about we meet in the middle and say 16?

From a perspective of most workloads e.g. asking for blockchain data, bitcoind (and electrs) acts very much like database software, e.g. posgresql. Postgresql is an interesting case because it utilizes process-based model - meaning it will spawn not just a thread, but a whole process for every client connection - waaay heavier than just a thread.

Even for that heavy model, the recommendations will be to limit it around around 500.

It takes many threads to saturate modern IO device. Just an example from a yt video I had on hand benchamarking filesystems. Each bar is number of workers. And these are dedicated IO-generating workers, generating heavy load in a tight loop. So in this benchmark, it will take like 4 workers. Possibly one worker is even more threads. But an rpc thread in bitcoind by necessity will be spending lots of time reading things and writing stuff to a tcp connection, doing some checks, re-encodings and what not. To actually get a maximum IO performance there will need to be a lot of them. I'd say 64 per core.

On my mostly idle laptop at this very moment there are 2k threads running:

On a modern Linux a thread is just a stack, which due to CoW optimizations will really amount to 1 or 2 pages (4K page size) of physically used memory plus a tiny (1K?) amount to track it all in the kernel.

There's really not much to save on thread counts in a grand scheme of things.

Personally, I like to run as lots and lots of tests in parallel, especially any sort of integration and e2e tests, which tend to have lots of dead time - waiting for stuff (like disk and network IO - latency-bound things). The usual limit to this is the available memory. Additional benefit is also that it tends to expose problems more often, as all operations tend to have more ... timing jitter.

So I'm really surprised that the default in bitcoind is 4. That's too little even for an old generation raspberry pi.

CONTRIBUTING.md

NIX.md

matthiasdebernardini · 2024-02-06T02:41:13Z

Can you have it print at the beginning, "First time takes a good long while..." or something like that. Some parts seem so slow that you'd think the shell crashed or something like that. Would be good for the user to know this.

Also I am having 8 tests fail, is that just me?

running 8 tests
test mempool_re_emits_if_tx_introduction_height_not_reached ... FAILED
test mempool_avoids_re_emission ... FAILED
test no_agreement_point ... FAILED
test mempool_during_reorg ... FAILED
test tx_can_become_unconfirmed_after_reorg ... FAILED
test test_sync_local_chain ... FAILED
test ensure_block_emitted_after_reorg_is_at_reorg_height ... FAILED
test test_into_tx_graph ... FAILED

failures:

---- mempool_re_emits_if_tx_introduction_height_not_reached stdout ----
Error: JSON-RPC error: transport error: Couldn't connect to host: Resource temporarily unavailable (os error 35)

---- mempool_avoids_re_emission stdout ----
Error: JSON-RPC error: transport error: Couldn't connect to host: Resource temporarily unavailable (os error 35)

---- no_agreement_point stdout ----
Error: JSON-RPC error: transport error: Couldn't connect to host: Resource temporarily unavailable (os error 35)

---- mempool_during_reorg stdout ----
Error: JSON-RPC error: transport error: Couldn't connect to host: Resource temporarily unavailable (os error 35)

---- tx_can_become_unconfirmed_after_reorg stdout ----
Error: JSON-RPC error: transport error: Couldn't connect to host: Resource temporarily unavailable (os error 35)

---- test_sync_local_chain stdout ----
Error: JSON-RPC error: transport error: Couldn't connect to host: Resource temporarily unavailable (os error 35)

---- ensure_block_emitted_after_reorg_is_at_reorg_height stdout ----
Error: JSON-RPC error: transport error: Couldn't connect to host: Resource temporarily unavailable (os error 35)

---- test_into_tx_graph stdout ----
getting new addresses!
got new addresses!
mining block!
Error: JSON-RPC error: transport error: Couldn't connect to host: Resource temporarily unavailable (os error 35)


failures:
    ensure_block_emitted_after_reorg_is_at_reorg_height
    mempool_avoids_re_emission
    mempool_during_reorg
    mempool_re_emits_if_tx_introduction_height_not_reached
    no_agreement_point
    test_into_tx_graph
    test_sync_local_chain
    tx_can_become_unconfirmed_after_reorg

test result: FAILED. 0 passed; 8 failed; 0 ignored; 0 measured; 0 filtered out; finished in 24.35s

error: test failed, to rerun pass `-p bdk_bitcoind_rpc --test test_emitter`
(nix:nix-shell-env) matthiasdebernardini@Matthiass-MacBook-Pro-2:~/Projects/bdk_nix$

flake.nix

dpc · 2024-08-07T17:39:12Z

Also, if you are using single bitcoind instance for all tests, you might want to increase the queue size and rpc thread count on it. Otherwise bitcoind randomly rejects connections, even when there isn't all that many of them yet. Though to that I'd add that the production code should probably do some retries as well.

Anyway, I'm guessing. Unfortunately non trivial tests that involve lots of different software pieces etc. often require some debugging and refactoring/tweaking to work reliably.

ordinals/ord#3364 (comment)

https://github.com/fedimint/fedimint/blob/1ca5227e0ceb8c4f3b0ca122364653e0f017a38c/devimint/src/cfg/bitcoin.conf#L9

8bf8c7d chore: fix clippy lints (Jose Storopoli) Pull request description: ### Description Fix some clippy CI lints ### Notes to the reviewers More caught by the Nix CI in bitcoindevkit#1320. ### Changelog notice chore: clippy lints ### Checklists #### All Submissions: * [x] I've signed all my commits * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md) * [x] I ran `cargo fmt` and `cargo clippy` before committing #### New Features: * [ ] I've added tests for the new feature * [ ] I've added docs for the new feature #### Bugfixes: * [ ] This pull request breaks the existing API * [ ] I've added tests to reproduce the issue which are now passing * [ ] I'm linking the issue being fixed by this PR ACKs for top commit: notmandatory: ACK 8bf8c7d Tree-SHA512: 6b53cb739e506d79106a2f42aa2b8fa28ef226543fbbf100225f10ed82257f6fd0218f09c0f1291803fbc9c1c88e32ba1c7e02fe3f601ccc9dc5be8a6efea6e1

oleonardolima · 2024-08-09T12:26:37Z

Try setting the ulimit to higher numbers. 1024 is still quite small for a lot of things running concurrently.

I tested with ulimit size up to 16384, and it still fails. I would guess it's something other than the ulimit.

Also, if you are using single bitcoind instance for all tests, you might want to increase the queue size and rpc thread count on it. Otherwise bitcoind randomly rejects connections, even when there isn't all that many of them yet. Though to that I'd add that the production code should probably do some retries as well.

I did a little bit of it and further testing, the main problem occurs in the bdk_testenv::TestEnv::mine_blocks, it's the same lineno failure for all the tests. Specifically when it's trying to first mine some blocks, and calling the RPC command generatetoaddress.

It's really weird, because I checked it and it's running the bitcoind on different ports, datadirs, and such 🤔.

https://github.com/fedimint/fedimint/blob/1ca5227e0ceb8c4f3b0ca122364653e0f017a38c/devimint/src/cfg/bitcoin.conf#L9

I think we should give it a try.

oleonardolima · 2024-08-13T20:29:52Z

https://github.com/fedimint/fedimint/blob/1ca5227e0ceb8c4f3b0ca122364653e0f017a38c/devimint/src/cfg/bitcoin.conf#L9

I think we should give it a try.

FWIW, I'm not sure if I did something wrong, but it still didn't solve the issue 🤔

storopoli · 2024-09-10T10:38:48Z

rebased to latest master.
Added the new checks in rust stable for the example-crates.
Once #1599 is merged I can delete the 3507055, since it is the same commit

bc83e41 fix: typos (Jose Storopoli) Pull request description:  ### Description more typos caught up during rebasing of #1320. ### Notes to the reviewers ### Changelog notice   ### Checklists #### All Submissions: * [x] I've signed all my commits * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md) * [x] I ran `cargo fmt` and `cargo clippy` before committing #### New Features: * [ ] I've added tests for the new feature * [ ] I've added docs for the new feature #### Bugfixes: * [ ] This pull request breaks the existing API * [ ] I've added tests to reproduce the issue which are now passing * [ ] I'm linking the issue being fixed by this PR ACKs for top commit: LagginTimes: ACK bc83e41 oleonardolima: ACK bc83e41 thunderbiscuit: ACK bc83e41. Tree-SHA512: 1dc76d64ddb0273c60899a6617dcbc63fab75057447080260b7050bb5d178ef56c6f79f2c7f1ca5b7b4eb09e2b1e67d782be91598ab58e6a3b3179d6b5edf5c5

storopoli · 2024-09-11T16:58:46Z

rebased to master after #1599 was merged

thesimplekid · 2024-09-11T17:29:33Z

I stole this for cdk just fyi been working well. https://github.com/cashubtc/cdk/blob/main/flake.nix

storopoli · 2024-09-11T19:52:50Z

rebased master after #1561 was merged

notmandatory

Overall this looks like a great use of Nix to setup a local dev env use the pre-checks to prevent common commit problems.

I would like to figure out the macos test threads issue, but that can be done in a follow-up PR. I also have a few other comments below.

Thanks all the hard work getting this to work.

Nix.md

.github/workflows/cont_integration.yml

flake.nix

.github/workflows/update_flake_lock.yml

Nix.md

README.md

justfile

storopoli · 2024-09-16T18:37:45Z

Thanks all the hard work getting this to work.

My pleasure.

Ok the trick that @thesimplekid has that does not need a Cargo.lock seems to work. Thanks! (I did some extra stuff that calls cargo update in the non-MSRV shellHooks)
I'll squash everything into a single commit and ask for a new round of reviews.

PS: Since we are changing the GitHub CI workflow names, we might need to update the branch protection rules for master after this gets merged.

notmandatory · 2024-09-25T02:55:07Z

I just ran into an error with the pre-push hook that it didn't think 2cf46a2 is signed because (I think) it was signed with an ssh key instead of pgp. Can you repro this @storopoli ? I found it when pushing latest master to my github fork repo.

You can switch flawlessly from Rust stable, MSRV, and nightly using either: - `nix develop .`: Rust MRSV - `nix develop .#msrv`: Rust MRSV - `nix develop .#stable`: Rust latest stable - `nix develop .#nightly`: Rust Nightly Also you don't need to worry about installing binaries from `bitcoind` or `electrs` crates, since we fetch these from `nixpkgs` (with pinned versions) and add them to your local environment. Additionally, all the extra dependencies that you need for WASM are handled as well. Finally, no need to set ENV vars: they are also set all by default to you. If you are committing typos, or failing to adhere to our guidelines in [CONTRIBUTING.md](CONTRIBUTING.md); we won't let you commit or push. This is handled by Nix's automated `pre-commit` hooks which gives helpful error messages. Please check the [Nix.md](Nix.md) file to learn more about Nix and what we are aiming to do with the Nix integration into BDK.

storopoli · 2024-09-27T19:51:37Z

You are hitting this: https://blog.dbrgn.ch/2021/11/16/git-ssh-signatures/

You need to whitelist @ValuedMammal's ssh key in error: gpg.ssh.allowedSignersFile needs to be configured and exist for ssh signature verification

notmandatory · 2024-10-01T14:58:29Z

@storopoli sorry to have to bump this one from 1.0 but I think we need to have some more discussions about how nix helps the project and where/how it should be introduced. Also there are multiple new things being introduced here and probably better to put ci, hooks, justfile changes in new prs. But when you're free let's schedule a time for a live that. Thanks!

storopoli · 2024-10-01T15:26:27Z

Sure, I'll move this to a draft for now.

storopoli force-pushed the js/nix-simple branch 7 times, most recently from ab132c9 to d139091 Compare February 4, 2024 18:50

dpc approved these changes Feb 4, 2024

View reviewed changes

storopoli force-pushed the js/nix-simple branch from d139091 to 710919e Compare February 4, 2024 19:09

notmandatory reviewed Feb 4, 2024

View reviewed changes

crates/bdk/Cargo.toml Outdated Show resolved Hide resolved

storopoli force-pushed the js/nix-simple branch from a1997cf to a5cce86 Compare February 4, 2024 19:42

storopoli requested a review from notmandatory February 4, 2024 19:42

storopoli force-pushed the js/nix-simple branch from a5cce86 to f141cd1 Compare February 4, 2024 19:53

oleonardolima reviewed Feb 5, 2024

View reviewed changes

CONTRIBUTING.md Show resolved Hide resolved

CONTRIBUTING.md Show resolved Hide resolved

NIX.md Outdated Show resolved Hide resolved

NIX.md Outdated Show resolved Hide resolved

dpc mentioned this pull request Feb 5, 2024

Default rpcthreads value (4) is too small bitcoin/bitcoin#29386

Open

1 task

storopoli force-pushed the js/nix-simple branch from f141cd1 to 0ceb17a Compare February 5, 2024 19:28

oleonardolima approved these changes Feb 5, 2024

View reviewed changes

This comment was marked as resolved.

Sign in to view

storopoli force-pushed the js/nix-simple branch from 0ceb17a to dd1754c Compare February 6, 2024 11:27

yanganto reviewed Feb 8, 2024

View reviewed changes

flake.nix Outdated Show resolved Hide resolved

yanganto reviewed Feb 8, 2024

View reviewed changes

flake.nix Outdated Show resolved Hide resolved

storopoli mentioned this pull request Aug 13, 2024

discussion: should we adopt the Cargo-minimal.lock and Cargo-recent.lock approach ? #1551

Open

storopoli force-pushed the js/nix-simple branch from b1ca648 to f4f3fd3 Compare September 10, 2024 10:22

storopoli mentioned this pull request Sep 10, 2024

fix: typos #1599

Merged

8 tasks

storopoli force-pushed the js/nix-simple branch from f4f3fd3 to 5f41abf Compare September 10, 2024 10:36

storopoli force-pushed the js/nix-simple branch from 5f41abf to 3507055 Compare September 10, 2024 10:51

storopoli force-pushed the js/nix-simple branch from 3507055 to 8c8c1e4 Compare September 11, 2024 16:56

storopoli force-pushed the js/nix-simple branch from 8c8c1e4 to 9d56e82 Compare September 11, 2024 19:50

notmandatory requested changes Sep 15, 2024

View reviewed changes

storopoli force-pushed the js/nix-simple branch from 9d56e82 to 0267841 Compare September 16, 2024 18:17

storopoli force-pushed the js/nix-simple branch from bf151a9 to f2e839c Compare September 16, 2024 18:38

storopoli requested a review from notmandatory September 16, 2024 18:38

storopoli force-pushed the js/nix-simple branch from f2e839c to 4e37eb2 Compare September 27, 2024 19:46

notmandatory removed this from the 1.0.0-beta milestone Oct 1, 2024

storopoli marked this pull request as draft October 1, 2024 15:26

notmandatory mentioned this pull request Oct 1, 2024

Remove CI Caching step #1637

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: move to Nix (simpler, easy to maintain version) #1320

ci: move to Nix (simpler, easy to maintain version) #1320

storopoli commented Feb 4, 2024 •

edited

Loading

dpc commented Feb 4, 2024

dpc commented Feb 4, 2024

storopoli commented Feb 4, 2024

evanlinjin commented Feb 4, 2024

storopoli commented Feb 4, 2024

dpc commented Feb 4, 2024

evanlinjin commented Feb 5, 2024

dpc commented Feb 5, 2024 •

edited

Loading

matthiasdebernardini commented Feb 6, 2024

This comment was marked as resolved.

dpc commented Aug 7, 2024 •

edited

Loading

oleonardolima commented Aug 9, 2024

oleonardolima commented Aug 13, 2024

storopoli commented Sep 10, 2024 •

edited

Loading

storopoli commented Sep 11, 2024

thesimplekid commented Sep 11, 2024

storopoli commented Sep 11, 2024

notmandatory left a comment •

edited

Loading

storopoli commented Sep 16, 2024

notmandatory commented Sep 25, 2024

storopoli commented Sep 27, 2024

notmandatory commented Oct 1, 2024

storopoli commented Oct 1, 2024

ci: move to Nix (simpler, easy to maintain version) #1320

Are you sure you want to change the base?

ci: move to Nix (simpler, easy to maintain version) #1320

Conversation

storopoli commented Feb 4, 2024 • edited Loading

Description

Notes to the reviewers

Changelog notice

Checklists

All Submissions:

New Features:

Bugfixes:

dpc commented Feb 4, 2024

dpc commented Feb 4, 2024

storopoli commented Feb 4, 2024

evanlinjin commented Feb 4, 2024

storopoli commented Feb 4, 2024

dpc commented Feb 4, 2024

evanlinjin commented Feb 5, 2024

dpc commented Feb 5, 2024 • edited Loading

matthiasdebernardini commented Feb 6, 2024

This comment was marked as resolved.

dpc commented Aug 7, 2024 • edited Loading

oleonardolima commented Aug 9, 2024

oleonardolima commented Aug 13, 2024

storopoli commented Sep 10, 2024 • edited Loading

storopoli commented Sep 11, 2024

thesimplekid commented Sep 11, 2024

storopoli commented Sep 11, 2024

notmandatory left a comment • edited Loading

Choose a reason for hiding this comment

storopoli commented Sep 16, 2024

notmandatory commented Sep 25, 2024

storopoli commented Sep 27, 2024

notmandatory commented Oct 1, 2024

storopoli commented Oct 1, 2024

storopoli commented Feb 4, 2024 •

edited

Loading

dpc commented Feb 5, 2024 •

edited

Loading

dpc commented Aug 7, 2024 •

edited

Loading

storopoli commented Sep 10, 2024 •

edited

Loading

notmandatory left a comment •

edited

Loading